Papers with memory representations
Closed-Book Training to Improve Summarization Encoder Memory (D18-1)
Copied to clipboard
| Challenge: | Extractive summarization models pick words, phrases, and sentences from the source text to form a summary, while abstractive models generate (samples) words from a fixed-size vocabulary instead of copying from text directly. |
| Approach: | They propose to add an additional ‘closed-book’ decoder without attention and pointer mechanisms to the model to improve its memorization capabilities. |
| Outcome: | The proposed model outperforms the baseline model on the CNN/Daily Mail dataset in terms of ROUGE and METEOR metrics, and achieves higher scores in a test-only setup. |
Memory efficiency and resource-rational encoding in sentence processing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that language models need to be constrained in their use of working memory for context, the analogue to human working memory (WM). |
| Approach: | They propose to inject noise into hidden representations of Transformer-based LMs to capture constraint on memory encoding. |
| Outcome: | The proposed model improves alignment with human reading times and makes them more compressed and categorical. |